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THE USE OF PILOT RATING IN THE EVALUATION OF 


AIRCRAFT HANDLING QUALITIES 
By George E. Cooper 
Ames Research Center 
and 

Robert P. Harper, Jr. 
Cornell Aeronautical Laboratory 

SUMMARY 


Pilot rating scales and their use in assessing aircraft handling 
qualities are reviewed historically, and objections that have been raised to 
limitations of earlier scales are considered in the development of a revised 
scale. Terminology used in the evaluation of handling qualities is reviewed 
and new definitions are proposed to improve communication and international 
understanding. Of particular significance is the new definition of handling 
qualities, which emphasizes the importance of factors that influence the selec 
tion of a rating other than stability and control characteristics. 

The experimental use of pilot rating is discussed in detail, with special 
attention devoted to (1) clarifying the difference between mission and task, 
(2) identifying what the rating applies to, (3) considering the pilot's 
assessment criteria, and (4) defining the simulation situation. The important 
elements of the report are then summarized in a suggested "Briefing Guide," 
designed for guidance in planning and executing handling qualities 
experiments . 


INTRODUCTION 


The widespread application of pilot rating scales in the evaluation of 
aircraft handling qualities has confirmed their basic utility, but has, at the 
same time, exposed some weaknesses of the scales as originally proposed. It 
was therefore considered desirable to re-examine existing rating scales with 
the purpose of developing a single improved scale and of clarifying its use in 
the evaluation of handling qualities. In response to an invitation from the 
Flight Mechanics Panel, AGARD, a paper entitled, "A Revised Pilot Rating Scale 
for the Evaluation of Handling Qualities," was prepared and presented at the 
September 1966 meeting (ref. 1). A longer version of this paper (ref. 2) con- 
tains some additional explanatory discussion of the use of pilot rating scales 
Additional constructive criticism was then based on experience gained with 
this revised scale. 



In general, the revised scale was preferred over earlier scales, but 
constructive criticism from many research and development groups also 
indicated the need for additional changes or clarifying discussion. Diffi- 
culties, for example, were experienced with the semantics in that certain 
words had rather different connotations in the United States, England, and in 
France. The purpose of this report, then, is to clarify and modify, as appro- 
priate, the material presented in references 1 and 2. One of the first objec- 
tives of the present report is to define precisely the basic terminology and 
explain the new features in the scale. The report goes on to discuss the more 
important factors that are considered by the pilot in the selection of a rat- 
ing or that will otherwise influence the rating. Throughout the discussion, 
attention is directed to the questions: (1) what is the pilot being asked to 

rate?, and (2) how will the experimental results be used? The answers to both 
questions have important bearing on the interpretation of evaluations made by 
pilots with different backgrounds, experience, and points of view. The final 
section of the discussion is devoted to a review of certain other consider- 
ations that are helpful in the design of handling qualities experiments and to 
the use of pilot rating. The important elements of the report are summarized 
in a condensed ’’Briefing Guide” for use in planning and executing handling 
qualities experiments . 


DISCUSSION OF HANDLING QUALITIES 


Clarification of Terms 

For a pilot rating scale to be universally acceptable and consistently 
applied in the evaluation of handling qualities, the terminology must be 
easily understood by all persons working in the field. Those terms requiring 
specific attention are defined in appendix A, and several definitions are sug- 
gested that may help clarify and standardize the terminology. Some of the 
terms suggested in reference 3 have been adopted in this report. Those most 
significant to a discussion of handling qualities are examined in considerable 
detail in the following paragraphs. Others are discussed as the need arises. 

Handling qualities .- The term ’’Handling Qualities” requires a clear 
definition in order to emphasize that it includes more than just stability and 
control characteristics. Other factors that influence the handling qualities 
are the cockpit interface (e.g., displays, controls), the aircraft environ- 
ment (e.g., weather conditions, visibility, turbulence) and stress, the 
effects of which cannot readily be segregated. Thus in most tests, handling 
qualities are really being evaluated in the aggregate. 

In appendix A, ’’Handling Qualities” is defined as ’’those qualities or 
characteristics of an aircraft that govern the ease and precision with which 
a pilot is able to perform the tasks required in support of an aircraft role.” 
The generally accepted meaning of ’’Flying Qualities” is similar to this defi- 
nition of ’’Handling Qualities,” so only the latter term is used in this 
report . 
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Figure 1 illustrates the factors besides stability and control that 
influence handling qualities. Here the primary elements of the pilot control 
loop are arranged to illustrate their relationship to the operation of the 
pilot-vehicle combination. In addition to the pilot, the task, and the 
stability and control characteristics, factors shown as influencing closure of 
the pilot control loops are the cockpit interface, the aircraft environment, 
and the pilot's stress. 



Figure 1.- Elements of control loop that influence handling qualities. 


Mission.- The term "mission" has been used in the United States rather 
loosely, and may actually have several meanings, depending on how it is used. 
"Mission" has been used to identify, in a general sense, the purpose or 
objective for which an aircraft is built. It has also been used to designate 
a complete flight or sortie or even an undefined part of the flight. By 
inference, this undefined part is usually the special flight phase during 
which the primary assignment is carried out. 

To avoid this ambiguity, the terms "role," "flight" or "sortie," "flight 
phase," and "flight subphase" are defined for use in place of "mission" (or 
mission element) . The continued use of "mission" in relation to handling 
qualities, however, makes it worthwhile to have an acceptable definition. 

One that has been suggested (ref. 4) is "the composite of pilot-vehicle func- 
tions that must be performed to fulfill operational requirements." In the 
present report, "mission" is a general term used to convey this concept of 
"operational requirements," that is, the objectives or delineation of what it 
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is that the pilot-vehicle combination must be able to accomplish. As a rule, 
the mission objectives (required operations) are cited in more detail as the 
flight segment (flight phase or subphase) of interest becomes more specific. 

The distinct differences intended between "mission" and "task" will be 
clarified in the following paragraphs. 

Task . - The term "task" also has various connotations. We are concerned 
here only with the pilot's task, which includes controlling the aircraft as 
well as associated functions, not directly related to controlling the aircraft, 
such as navigation and communications. A task in the sense that it is used in 
handling qualities evaluations is defined as "the actual work assigned a pilot 
to be performed in completion of, or as representative of, a designated flight 
segment." In being representative of a flight phase, for example, the impor- 
tant pi lot- vehi cle functions required to fulfill the operational requirements 
for that flight phase would be represented in the task. Use of "task" and 
"mission" differs then, in that a task represents what the pilot is actually 
asked to do (as in a simulation task) while a mission refers to all opera- 
tional requirements the pilot-vehicle combination must be able to accomplish 
if the "intended use" of the aircraft is to be fulfilled. 

It is convenient to consider the complete task to be composed of (1) the 
control task, and (2) auxiliary tasks. The control task requires actuation 
of the principal controls and the selectors as required. The auxiliary tasks 
involve the pilot in actions other than direct control of the aircraft. 

Flight phase .- The terms "flight," "flight phase," and "flight subphase" 
denote the flight profile of an aircraft and its subdivision into convenient 
segments. The delineation of aircraft role and the division of a complete 
flight into discrete segments for more definitive examination is illustrated 
in figure 2. Representative examples of what is meant by aircraft role, com- 
plete flight, flight phase, and subphase are given in this figure as well as 
in appendix A. A subphase is defined as "that part of a flight phase having 
a single objective, and a single configuration or change in configuration." 

A subphase evaluation then would provide a direct correlation between a 
specific set of stability and control parameters and pilot rating. 

In summary then, the role of an aircraft defines its intended use only 
in a general sense. The mission delineates this use in terms of specific 
objectives, that is, the required operations of the pi lot-vehicle combination. 
The task delineates those aspects of the mission that are work assigned to the 
pilot. 


Methods of Determining Aircraft Handling Qualities 

The relationship between stability and control parameters and the degree 
of suitability of the airplane for the mission may be examined by: 
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1. Theoretical analysis 

2. Experimental performance measurement 

a. Pilot input 

b. Pilot-vehicle output 

3. Pilot evaluation 

Each approach has an important part in the complete evaluation. One 
might ask, however, "Why is the pilot assessment necessary?" The answer must 
consider the two alternatives, theoretical analysis and performance measure- 
ment. At present, the applicability of the mathematical analysis including 
representation of the human operator is restricted to the analysis of speci- 
fic simple tasks. Since the intended use (mission) is made up of several 
tasks and several modes of pilot-vehicle behavior, it is difficult first to 
describe accurately all modes analytically, and, second to integrate the qual- 
ity in the separate tasks into a measure of overall quality for the intended 
use. Theoretical analysis is fundamental to the analytical prediction of 
handling qualities, but cannot adequately treat the complex interactions that 
are now investigated by means of experimental pilot evaluation. 

The attainment of satisfactory performance in fulfilling a designated 
mission is, of course, a fundamental reason for our concern with handling qual- 
ities. Why, then, cannot the experimental measurement of performance replace 
pilot evaluation? Why not measure pilot-vehicle output performance in the 
intended use? Isn't good performance consonant with good quality? Unfortu- 
nately, the answer to the latter question is "not always." 
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A significant difficulty arises here in that, first, the tasks selected 
for measuring performance may not demand of the pilot all that the real mis- 
sion demands, especially in terms of distractions, auxiliary tasks, and pilot 
stress. In the second place, pilot performance must also be measured and 
interpreted so that the pilot-vehicle performance can be evaluated correctly. 
The pilot is an adaptive controller whose goal (when he is so instructed) is 
to achieve good performance. In a specific task, he is capable of attain- 
ing essentially the same performance for a wide range of vehicle characteris- 
tics, at the expense of significant reductions in his capacity to assume other 
duties and to plan subsequent operations. Significant differences in his task 
performance may not be measured when very real differences in mission suit- 
ability do exist. The pilot's performance must be measured and analyzed 
properly to show these differences, and one must have a clear appreciation of 
what constitutes objectionable inputs. 

In the third place, it is difficult, if not impossible, at the present 
time to measure all important aspects of pilot performance. Encouraging 
results have been obtained in specific instances (refs. 5-7) wherein good cor- 
relation has been obtained between measurements of the physical effort 
exerted by the pilot (i.e., integral of pilot control displacement, force, 
etc.) and pilot rating. In such cases, it must be assumed that differences in 
mental effort and attention were not significant. Rather than attempt to pre- 
judge the influence of mental effort and attention, the term workload is 
defined to include both mental and physical effort. The use of modifying 
terms will be necessary then when a distinction is necessary between mental 
and physical workload. 

The questions that arise in using performance measurements may be 
summarized as follows: (1) For what maneuvers and tasks should measurements 

be made to insure fulfillment of the mission objectives? (2) How do we inte- 
grate and weigh the performance in several tasks to get an overall measure of 
quality if measurable differences do exist? (3) Is it necessary to measure or 
evaluate pilot workload for the performance to be meaningful? If so, how are 
these factors weighed with those in (2)? (4) What disturbances and distrac- 

tions are necessary to provide a realistic workload for the pilot while his 
performance in a specified task is being measured? The difficulties encoun- 
tered in answering each of these questions provide some of the reasons why con- 
tinued reliance upon pilot evaluation is necessary and why much of the 
detailed discussion is included in this report. Although the use of measured 
task performance is not discussed further in this report, continued efforts 
to measure and interpret both the pilot performance and the pilot-vehicle 
performance should be encouraged. Such information is important to the under- 
standing of pilot adaptation or "learning curves," and to the interpretation 
of pilot evaluation data. 

Pilot evaluation still remains the only method of assessing the 
interactions between pilot-vehicle performance and total workload in determin- 
ing suitability of an airplane for the mission. It provides a basic measure 
of quality and serves as a standard with which pi lot -airplane system theory 
may be developed, performance measurements may be correlated, and significant 
airplane design parameters and characteristics may be determined and 
correlated. 
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Pilot evaluation data generally consists of: (1) the pilot rating, or 

shorthand representation of the flying characteristics as they relate to mis- 
sion accomplishment, and (2) the pilot comments that identify those 
characteristics that interfere with the intended use. 

A pilot rating is a portion of the technical report of the evaluator, and 
is the overall summation of the suitability of the vehicle for the specified 
use. The pilot rating scale is then a systematic means of denoting the qual- 
ity of the pilot-vehicle combination in the accomplishment of its intended 
purpose. 


EARLY RATING SCALES 


In early handling qualities research, each investigator tended to develop 
a rating scale peculiar to the needs of his specific program, or to modify an 
existing one. With experience, certain pilot rating scales proved successful 
and stimulated further interest in them. Two such early scales are discussed 
here . 


In reference 8 the original Cooper Scale (fig. 3) proposed the basic 
framework of boundaries that is still the foundation of most pilot rating 
scales, including the presently proposed one. This framework involves several 
grades of quality pertaining to the intended use of the vehicle. 

Acceptable and Satisfactory - sufficiently good 

Acceptable but Unsatisfactory - not sufficiently good, but still usable 

Unacceptable - not usable for mission 

Uncontrollable 


r 

Ad jective 
rating 

Numerical 

rating 

Description 

Primary mission 
accomplished 

Can be ^ 
landed 



i 

Excellent, includes optimum 

Yes 

Yes 

NORMAL 


2 

Good, pleasant to fly 

Yes 

Yes 

OPERATION 

Satisfactory 

3 

Satisfactory, but with some mildly 






unpleasant characteristics 

Yes 

Yes 



4 

Acceptable, but with unpleasant 






characteristics 

Yes 

Yes 

EMERGENCY 


5 

Unacceptable for normal 



OPERATION 

Unsatis factory 


operation 

Doubtful 

Yes 



6 

Acceptable for emergency 






condition only * 

Doubtful 

Yes 



7 

Unacceptable even for 



NO 



emergency condition* 

No 

Doubtful 

OPERATION 

Unacceptable 

8 

Unacceptable - Dangerous 

No 

No 



9 

Unacceptable - Uncontrollable 

No 

No 



10 

"Motions possibly violent enough to 



V 

Unprintable 


prevent pilot escape" 


J 


* Failure of stability augmenter 


Figure 3.- Original Cooper Rating Scale. 
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Important contributions of the early scales were to emphasize pilot acceptance 
of handling qualities with respect to the intended use by: 

a. Encouraging investigators to define adequately the program objectives 
and the intended use of the aircraft. 

b. Firmly establishing within the scale the acceptable-unacceptable, and 
satisfactory -unsatisfactory boundaries . 

c. Creating a logical basis that would enable the pilot to express his 
assessment accurately and consistently. 

The original Cooper Scale included too many different concepts. The 
introduction of stability augmentation and undefined failure modes, of normal 
and of emergency operation, and the separation of the landing task from the 
primary mission led to ambiguous interpretations. 


The CAL scale of figure 4 was 
developed primarily because the 
Cooper Scale was confusing to some in 
that it could be interpreted as intro- 
ducing an alternate mission concept. 
Separate boundaries were shown in the 
Cooper Scale for normal operation and 
for an undefined emergency condition. 
By removing this doubt of mission 
completion in the adjective descrip- 
tions in the acceptable range, as 
well as removing all consideration of 
an alternate mission from the scale 
itself, the CAL scale clarified this 
situation. However, the very simple 
descriptions of the CAL scale are not considered particularly helpful by many 
pilots and the dual use of "fair" and "bad" was confusing to some. 

Having now identified certain deficiencies and objections to two rating 
scales and clarified some of the terminology, we are in a position to 
construct a revised scale. 


r 


\ 


Adjective description 

Numerical 

Category 

within category 

rating 

Acceptable 

Excellent 

1 

and 

Good 

2 

satisfactory 

Fair 

3 

Acceptable 

Fair 

4 

but 

Poor 

5 

unsatisfactory 

Bad 

6 


Bad* 

7 

Unacceptable 

Very bad** 

8 


Dangerous 1 

9 

^ Unflyable 

Unflyable 

10 J 


*Requires major portion of pitot's attention 
**Controtlable only with a minimum of cockpit duties 
'Aircraft just controllable with complete attention 


Figure 4.- CAL Rating Scale. 


REVISED PILOT RATING SCALE 


Major Categories 

Category selection .- The Cooper and CAL Rating Scales and the revised 
scale proposed in references 1 and 2 have the same basic structure. The major 
categories are identified as "satisfactory" and "unsatisfactory," or "accept- 
able," "unacceptable," and "uncontrollable." Such terms relate only to the 
individual pilot's assessment of quality relative to the intended use of the 
aircraft. The intention has been to encourage the pilot to make the important 
decisions identified with these terms. These categories are systematically 
arranged so that the pilot can choose between two alternatives that lead to 
the proper category. 
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This structure is illustrated by 
the flow chart in figure 5, which 
enables one to trace the series of 
dichotomous decisions the pilot makes 
in arriving at the final rating. As 
a rule, the first decision is fairly 
obvious. Are the handling qualities 
controllable or uncontrollable? To 
determine whether this decision 
applies throughout the task or flight 
phase and in context of the defined 
role may not be so obvious. 


Figure 5.- Sequential pilot-rating decisions. If the Vehicle is uncontrollable 

it is rated 10. If it is control- 
lable, the second decision is whether it is acceptable or unacceptable. If 
unacceptable, the rating 7, 8, or 9 will be selected (rating 10 has been 
excluded by the ’’control lable M answer to the first decision). If it is accept 
able, the third decision is whether it is satisfactory or unsatisfactory. If 
unsatisfactory, the rating 4, 5, or 6 will be selected; if satisfactory, the 
rating 1, 2, or 3 will be selected. 


Category definitions .- The dichotomous decisions outlined in the previous 
paragraph logically lead to four categories of quality, some of which have 
been difficult to describe accurately by a single adjective. There are objec- 
tions to the designation "unsatisfactory but acceptable" and "unacceptable but 
controllable," but no simple adjective descriptions for these categories have 
been found that will satisfy everyone. In spite of strong emphasis as to the 
need to establish clearly that the terms are related to the pilot's own 
assessment of acceptability and not to any existing standards, specifications, 
or other Acceptance Criteria, objection to "acceptable" and "unacceptable" in 
the scale remains. 


Regardless of the terms used in a revised rating scale, it is necessary 
to clarify the intended meaning of satisfactory, unsatisfactory, acceptable, 
tolerable, unacceptable, controllable, and uncontrollable. Let us examine, 
first, what is meant by "controllable," even though it has caused less diffi- 
culty than the other terms. To control is to exercise direction of, to com- 
mand, or to regulate. The determination as to whether the airplane is 
controllable must be made within the framework of the defined mission or 
intended use. An example of the considerations of this decision would be the 
evaluation of fighter handling qualities during which the evaluation pilot 
encounters a situation in which he can maintain control only with his complete 
and undivided attention. The vehicle is "controllable" in this situation in 
the sense that the pilot can maintain control only by restricting the tasks 
and maneuvers he is called upon to perform and by giving the configuration his 
undivided attention. However, for him to answer, "Yes, it is controllable in 
the flight phase (or task)," he must be able to retain control in all mission 
oriented tasks and other required operations without sacrificing effort and 
attention to his overall duties. 
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Consider now the meaning of acceptable, the usage of which has been the 
subject of some controversy, and the alternate suggested terra "tolerable.” 

The dictionary shows "acceptable" to mean that a thing offered is received 
with a consenting mind; "unacceptable" to mean that it is refused or rejected. 
Acceptable means that the flight phase (or task) can be accomplished; it 
means that the evaluation pilot would agree to use it for the designated role; 
that such deficiencies as may exist can be endured or tolerated. Use of the 
term "acceptable" does not say how good it is, but it does say the pilot con- 
siders it good enough for the intended use. With these characteristics, the 
flight phase (or task) can be accomplished with adequate precision. The task, 
for example, may be accomplished with considerable effort and concentration 
on the part of the pilot, but the level of workload required to achieve this 
performance is tolerable and not unreasonable in context with the intended 
use. By the same token, "unacceptable" does not necessarily mean that the 
designated flight phase (or task) cannot be accomplished; it does mean that 
the necessary performance cannot be achieved or that the effort, concentration, 
and workload required are of such magnitude that the evaluation pilot rejects 
the aircraft for this phase of its intended use. 

Consider now a definition of "satisfactory." The dictionary defines this 
as adequate for the purpose, of a kind to meet all requirements or expecta- 
tions. A pilot ’s definition of satisfactory might be that it isn T t neces- 
sarily perfect, or even good, but it is good enough that he wouldn’t ask that 
it be changed. It meets a standard; it has sufficient goodness; it’s of a 
kind to meet all pilot demands for the intended use. 

Unsatisfactory implies that there is insufficient goodness to meet all 
pilot demands; that it has deficiencies and objectionable characteristics 
which he feels should be corrected. Unsatisfactory includes all that is not 
satisfactory, just as acceptable includes all that is not specifically refused 
as unacceptable. A specific category is then that which is "unsatisfactory 
but acceptable." This category has previously been referred to by the shorter 
term "acceptable" but could also be referred to as "tolerable" in the sense, 
"capable of being borne, supportable, bearable." As a result of the possibil- 
ity of the misinterpretation in the use of "acceptable" it has been found 
preferable to use "tolerable" rather than "acceptable" in the rating scale. 

Thus, the quality is either: 

a. Satisfactory - good enough without improvement, and, therefore, of 
the best category, or 

b. Unsatisfactory but tolerable - just good enough, adequate for the 
purpose but improvement desirable, and, therefore, of the next best category, 
or 

c. Unacceptable to the pilot - not suitable for the purpose but still 
controllable, and in the third category, or 

d. Uncontrollable - unacceptable for the purpose and of the poorest 
quality and in the fourth category. 
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Inasmuch as only four categories of quality are needed to identify and 
describe handling qualities, it is possible to simplify the dichotomous 
decision process illustrated in figure 5 and eliminate "acceptability” terms. 

By considering the following three decisions the pilot will arrive at one 
of the four categories previously discussed: 

1. Is the vehicle controllable? 

2. Is adequate performance attainable with a tolerable workload? 

3. Is the vehicle satisfactory without improvement? 


f ADEQUACY FOR SELECTED TASK OR REQUIRED OPERATION ) 



1 Desired or cleorly odecpate 
performance is attainable 
» No significant deficiencies 
i Good enough without improvement 


> Control will be lost during some 
portion of required operation. 

> Improvement is mandatory 


> Adequate performance is attainable 
but requires pilot compensation. 

1 Has deficiencies which warrant 
improvement 

i Performance is inadequate even with 
maximum tolerable pilot compensation 
i Has major deficiencies which require 
improvement 


Figure 6.- Major category selection and definition. 


Being able to designate a 
category by a single adjective was 
considered less important than that 
the definition of each category be 
precise. The rating scale now pro- 
posed is therefore based on the cate- 
gory definitions provided in figure 6. 
These are reasonably concise and 
emphasize the pilot's decisions 
involved in category selection. 

Fundamental to this selection is 
the consideration of whether adequate 
performance is attainable in the 
selected task or required operation 
(mission) and whether or not defici- 
encies are present which require 
pilot compensation. 

Assessment of task performance 
and pilot workload then enables the 
pilot to decide whether the handling 
qualities are good enough without 
improvement, have deficiencies for 
which improvement is desired, or have 
deficiencies for which improvement is 
essential . 


Individual Ratings 

The complete revised rating scale 1 in figure 7 includes further 
subdivisions of quality within three of the four major categories with appro- 
priate descriptions for each numerical rating to define quality differences. 
These allow the pilot a sufficient range of handling quality descriptions for 
most situations. Repetition of descriptive terms from the category defini- 
tions has all but been eliminated in the individual rating descriptors. 

1 Small and large copies of figure 7 are available. Send requests to 
Technical Information Division, Ames Research Center, Moffett Field, Calif. , 
94035 . 
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Figure 7.- Handling Qualities Rating Scale. 


Considered desirable in simplifying the scale for the experienced user, this 
arrangement, however, makes it mandatory that any user understand and utilize 
the category definitions and make the decisions outlined on the left. The 
important "boundary" decisions between pilot ratings of 3 and 4, 6 and 7, or 
9 and 10 cannot be made by reference to the individual rating descriptors 
alone. It is emphasized that these descriptions supplement the sequential 
decisions that lead the evaluation pilot to the particular category within 
which the descriptions of the individual ratings are given. It was considered 
fundamental to a good, easily applied scale that the descriptions be both 
brief and general. Key words and phrases were sought that would easily be 
understood and yet sufficiently definitive so that each rating would be 
clearly separated from every other rating. The following paragraphs discuss 
and explain some of the factors related to the rating scale. 

Performance and workload .- Consideration was given to describing quality 
in terms of both performance (precision of aircraft control) 2 and workload 
(effort and attention) 2 for each numerical rating. Upon closer examination, 

2 When applied to handling qualities, the term "performance" alone is 
intended to mean the precision of aircraft control attained by the pilot, that 
is, the pi lot-vehicle performance. The term "workload" is intended to convey 
the amount of effort and attention, both physical and mental, that the pilot 
must provide to attain a given level of performance. "Pilot performance" may 
be used to describe the measure of physical workload (effort and attention) 
used in performing a task. 
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however, it was concluded that these factors were so interdependent and 
subject to tradeoff by a pilot that it would be impossible to define non- 
conflicting individual ratings. Precision of control could not be defined 
independently of the amount of effort and attention required of the pilot for 
any task . 

The best category is defined as desired, or clearly adequate performance 
in association with a satisfactory level of pilot workload. The second best 
category is defined by achievement of adequate performance although the pilot 
is required to compensate for deficiencies by increasing his workload. It is 
tolerable to him but he desires improvement. The third category involves 
major deficiencies because the pilot finds it impossible to achieve adequate 
performance even though his attempt to compensate increases the total workload 
to the maximum tolerable in context of the task or mission. Uncontrollable 
obviates further concern with performance, but as in all previous categories, 
must also be considered in the context of the selected task or required 
operation (mission) . 

Compensation.- "Pilot compensation" as used in the scale is intended to 
indicate that the pilot must increase his workload to improve aircraft perfor- 
mance. It relates the pilot's difficulty in completing a task with the 
precision required for that task. Stated another way, it is the measure of 
additional pilot effort and attention required to maintain a given level of 
performance in the face of less favorable or deficient characteristics. The 
total workload is then comprised of the workload due to compensation for air- 
craft deficiencies plus the workload due to the task. 

Referring to figure 6, we see that it is really only necessary to define 
that which constitutes adequate performance when answering the second question. 
The precision of control required for any task is most easily defined in terms 
of the end result obtained. For example, the approach performance can be spec- 
ified in terms of the threshold "gate," and landing performance is measured in 
terms of vertical velocity at touchdown and dispersion about the intended 
touchdown point. The necessity for holding airspeed, altitude, flight path, 
or other parameters within specified limits throughout each flight phase or 
subphase may also be specified, but the cause and effect of occasionally 
exceeding the limits may require pilot interpretation. The pilot must balance 
both aspects of performance - maintaining precision of control and the end 
result - against his own effort and attention in arriving at a rating. 

The category definitions of figure 6 recognize the interaction between 
performance achieved and the pilot workload required to compensate for defi- 
cient characteristics. In the best category, no significant compensation, in 
terms of added workload, is required by the pilot to achieve adequate perfor- 
mance. In fact, adequate performance is clearly achievable and it may just 
as easily be possible to attain some higher level of performance, such as 
might be designated broadly as desired performance. From the definition, it 
may be deduced that desired, or at least clearly adequate, precision of con- 
trol can be obtained with relative ease or a low level of effort and atten- 
tion. With deficiencies characterized by the second category, considerably 
more pilot effort is required and desired performance is not necessarily 
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obtained. Now, however, a lower level of handling performance, but one 
adequate for the purpose, can be achieved as long as the pilot will increase 
his workload, even to the maximum tolerable for the selected task. If major 
deficiencies exist, adequate precision of control would require an excessive 
level of pilot workload and even the maximum tolerable pilot workload will not 
enable achievement of the level of performance that is considered adequate 
for the selected task. In this category, the additional deficiencies that 
increase demands on the pilot can only be expected to result in inadequate 
performance, and the primary question becomes one of controllability. The 
manner in which precision of control may be traded off against pilot effort 
and attention is complex and defies explicit definition for all flight phases 
or specified tasks. However, such compensation is available to the pilot and 
is probably involved in the evaluation of handling qualities. 

Failures or emergency operation .- The revised scale contains no reference 
to failure considerations or emergency operation. In effect, this means that 
the pilot need not always concern himself during an evaluation with the prob- 
ability of a failure, nor with the length of time he might be faced with 
deficient handling qualities. The time (duration) during which particular 
characteristics must be coped with is inherently defined by the role and the 
flight phase or task being evaluated. For operation based on a normal air- 
craft state, the duration can be based explicitly on the aircraft role. If a 
failure occurs, however, it may be necessary to consider alternatives like 
aborting the flight or changing the flight plan to permit retreat to a more 
favorable flight condition; in either case, the length of time that the criti- 
cal flight condition must be tolerated might be shortened. Unless such 
alternatives are spelled out in the task definition, however, the pilot must 
always treat a failure state as having to be coped with for the duration of 
the task or flight phase, depending on which is being rated. Normal and 
emergency states then will require separate evaluations. 

There are environmental conditions as well as minor failures which can 
limit aircraft operation without being identified as an emergency. Excessive 
turbulence or crosswind could preclude attempting a landing and thus require 
that the flight plan be altered accordingly. Whenever limitations to the 
operation are indicated or accepted in lieu of specifying a critical task this 
fact should be clearly noted. 

Pilot skill .- The pilot effort and attention required for a given task or 
flight phase will, of course, depend somewhat upon individual pilot skill and 
state of training. In the early versions of the revised scale, the term 
"skill" was used in the 8 and 9 rating descriptions in consideration of the 
role of pilot skill in the degree of controllability which could be achieved. 
Further consideration, however, has led in the 8 and 9 rating descriptions 
to the substitution of phrases reflecting pilot workload required to even con- 
tinue the task and the relative difficulty in maintaining control, in order to 
retain a consistency and compatibility with the rest of the scale. In this 
manner, pilot skill becomes a consideration of the total program for which the 
evaluation is being made. 
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It is the opinion of the authors that as a general rule each pilot should 
judge the suitability of any set of airplane characteristics in terms of his 
own skill and training, and in terms of the required operations and circum- 
stances as defined in the experiment. The effects of differing skills should 
be determined from the results obtained from evaluation pilots of different, 
but representative, levels of experience and training. Exceptions to this 
general rule have occurred, however, when the research or development test 
pilot is asked to evaluate handling qualities with respect to his understand- 
ing of the lowest degree of skill and training existent in a group of 
operational pilots. 

Operating margins and safety.- The question of safety cannot be separated 
from handling qualities because the precision with which certain parameters 
are controllable determines whether available safety margins are apt to be 
exceeded. The margins provided, as well as the consequence of exceeding them, 
will certainly influence the effort and attention which a pilot applies. Oper- 
ating margins with respect to such parameters as airspeed, angle of attack, 

Mach number, and altitude then represent the constraints that define required 
precision of control, and thereby influence the rating selection. Operating 
margins and safety should be recognized as constraints upon handling qualities 
that are inextricably related to performance and workload. 


Considerations Associated With the Structure of the Revised Scale 

Why a 10-point scale?- In discussing the revised scale, one question 
that might be anticipated concerns the number of individual ratings the scale 
defines. Most simply, the number is related to the four categories already 
selected. Separating each of the upper three categories into three subratings 
appears to provide an adequate spread for pilot use. Additional ratings in 
the fourth category (uncontrollable) would not appear to be of general value. 

A change in the number of individual ratings for each category was not deemed 
necessary or desirable when considered in light of the large amount of exper- 
ience with the previous scales. 

Identifying the revised rating.- The oft -prod aimed criticism that the 
scale should start with 10 and progress to 1 instead of from 1 to 10 may be 
valid, but there are also examples that support the 1 to 10 logic. We are 
reluctant at this point to suggest a change simply because of the widespread 
use of 3-1/2 and 6-1/2 boundaries. To now reverse these would likely 
introduce considerable confusion and would not necessarily have long-range 
benefits . 

In proposing a revised scale, it was recognized that some confusion might 
result from continued use of the same numerical scale that has been identified 
with both the Cooper and CAL Scales. In references 1 and 2, the authors pro- 
posed a modified identification system using letters A and U in conjunction 
with the numbers 1-9. Subsequent experience and comment indicated this to 
be unnecessary, as long as the rating numerals are associated explicitly with 
the particular scale used. 
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Linearity.- This is a desirable characteristic of any scale. A 
temperature scale is linear with heat added for a material with constant spe- 
cific heat, in that the temperature rise per unit quantity of heat added is 
the same throughout the scale. Temperature is a normal and useful scale asso- 
ciated with comfort. Even though temperature may be quite linear with heat 
added, comfort is not linear with temperature. With what should the pilot rat- 
ing scale be linear? Since it is purported to measure quality, it should then 
be linear with the added quality of the pilot-vehicle combination in that the 
change in pilot rating per unit quality addition should be the same throughout 
the rating scale. The rating scale may possibly have this characteristic, but 
to demonstrate that the scale is indeed linear would require an independent 
measure of quality that does not presently exist. Since the basic merit of 
the scale is not significantly affected by the lack of demonstrated linearity, 
this factor has not been considered further. McDonnell in reference 4 
describes a study that establishes a correlation between the Cooper Scale, 
the revised scale of references 1 and 2, and a linear scale. 

Ordinal versus interval scale.- An interval scale is desirable, but the 
proposed pilot rating scale cannot be shown to be an interval scale. The 
authors have accepted it as being ordinal. It is, however, primarily an abso- 
lute scale rather than a relative one. The pilot rating is given for a con- 
figuration in the context of its acceptability to the pilot for the specified 
flight phase (or task) and not in terms of its goodness with respect to a con- 
figuration already evaluated. Fortunately, the concentration and effort 
required in performing each evaluation tends to suppress in the pilot's mem- 
ory the characteristics of preceding configurations, enabling him to consider 
objectively each configuration on its own merits for the required operations 
without continually making paired comparisons. Pilots are reluctant to rate 
something as excellent or optimum for fear that a subsequent configuration 
will be better than anything they considered possible. 

Words versus numbers.- The basic structure of the rating scale is 
completely dependent on words and their explicit definitions. The numeral 
associated with the evaluator's final decision is an expedient, a shorthand 
symbol. One risk associated with a numerical scale is that engineers will 
attempt to treat the pilot rating data with mathematical operations that are 
rigorously applicable only to a linear interval scale. Although some insight 
is sometimes gained, analysis of specific pilot rating data should not be 
totally dependent on such mathematical operations. 

Differing standards of acceptance for the same mission .- One difficulty 
that has arisen in the use of pilot rating data can be illustrated by the fol- 
lowing example. In an evaluation program for the landing approach flight 
phase of a commercial air transport the role and mission were carefully 
defined, the program was run, and the results were reported. One of the eval- 
uation pilots, an airline pilot, subsequently remarked that his airline would 
not accept any airplane with worse than a 4-1/2 rating for the landing 
approach. However, using the generally accepted interpretation of the rating 
scale, this pilot had said, as an evaluation pilot, that aircraft with worse 
characteristics which he rated at 5 and 6 could still be considered acceptable 
for the intended use. In the first case he was reflecting the viewpoint of 
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airline management and not speaking from the pilot ! s standpoint. It must be 
recognized that the final determination of overall acceptability of an air- 
craft for a given role may be influenced by factors other than just the 
piloting rating of the handling qualities. 

In deciding what to buy, any customer considers what he will get in terms 
of how much it will cost. It is easy to envision similar decisions being made 
to buy only that which is above the 3-1/2 boundary. And similarly, one can 
envision a reluctant decision to buy as low as a 6 rating, or to even accept 
a 7 or 8 rating (if the rating applied to a flight condition or aircraft fail- 
ure state of low probability), but only if all other possibilities for pur- 
chase of a better aircraft had been excluded. The basic pilot rating data on 
which these decisions are based, however, must be strictly mission-oriented if 
the subsequent quality versus cost decisions are to be meaningful. That is to 
say, the role must be understood and each flight phase adequately defined by 
the piloting task(s) provided. 

EXPERIMENTAL USE OF PILOT RATING OF HANDLING QUALITIES 


In previous usage of pilot rating scales, too little attention has been 
given to defining just what the pilot was rating and how the data were to be 
used. In defining the experiment and in reporting the results, it became 
apparent that (1) the term n handling qualities” requires a tighter, less ambig 
uous definition, (2) an accounting is required for certain factors in addition 
to stability and control, (3) a clearer understanding is needed of the differ- 
ence between mission and task definition, (4) certain considerations relative 
to the pilots assessment criteria must be understood, and (5) a simulation 
situation requires special definition and consideration of the use of the 
results and of the need for pilot extrapolation. 

In the following paragraphs attention is devoted to certain aspects of 
the design and execution of handling qualities experiments that are considered 
particularly important to the production of good data and the further 
clarification of the aforementioned problems. 


Program Development 

Fundamental to any handling qualities program is a clear definition of 
the primary objectives of the program and of the role and mission of the air- 
craft. Next is the designation of the tasks to be used in the course of the 
evaluation, and what the rating applies to. 

Program objective .- Generally, it is expected that the use of the data 
will conform reasonably well with the objectives outlined for the program. 

The primary objectives are related to either research, development, or accep- 
tance but there may also be a number of special considerations. (These 
include whether the program is exploratory or expected to be highly definitive 
etc.) It is essential to define whether stability and control characteristics 
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or cockpit interface elements are the primary variables of investigation and 
whether special conditions such as environmental disturbances, dynamic or 
static failures, or stresses are to be included in the task or considered in 
the rating. Of particular importance will be the selection of evaluation 
pilots and their indoctrination with respect to the program. The particular 
background, training, experience, and point of view of the evaluation pilots 
selected for the program may be determined by the program objective as well as 
by the intended use of the resultant data. 

Mission description .- The explicit description of the mission by 
delineation of the "required operations" is probably the most important con- 
tributor to the objectivity of the pilot evaluation data. The role must be 
carefully analyzed and a clear description and understanding reached between 
the engineer and the evaluation pilot as to their interpretation of the 
required operations. This description must include: 

a. What the pilot is required to accomplish with the aircraft, and 

b. The conditions or circumstances under which the mission is to be 
conducted . 

Because of their importance in specifying "what the pilot rating applies 
to" and in providing definitive guidance for the pilot in his evaluation, the 
"required operations" may often have to be given in considerable detail. In 
its simplest form, this consists of designating the flight phase or subphase(s) 
of interest, and including such variations as are considered critical and 
representative of actual operations. 

As an example of (b) , the conditions or circumstances might include 
instrument or visual flight, type of displays or controls in the cockpit, or 
other input information to assist the pilot in accomplishing the mission, etc. 
The environment in which the mission is to be accomplished must also be 
defined and considered in the evaluation, and could include, for example, the 
presence or absence of turbulence, day versus night, the frequency with which 
the mission has to be repeated, the preparedness of the pilot for the mission, 
and the pilot's level of proficiency. 

As noted under the discussion of the revised scale, it was recommended 
that steady-state failure considerations, with the attendant questions as to 
probability of occurrence, should be removed from the scale. This means that 
separate ratings would be obtained for each failure mode. Steady-state fail- 
ure modes would then normally be considered as existing throughout the evalu- 
ation task or flight phase, unless, of course, a specific evaluation program 
is designed to consider them otherwise. 

In evaluating the impact of transient disturbances caused by either 
environmental conditions or system failures, the pilot is still faced with the 
basic decision of what tradeoff he will accept between performance and work- 
load and how long he must cope with the condition. 

High probability occurrences, such as low to moderate turbulence, wind 
shear, and cross wind, can be rated separately, as if occurring throughout the 
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task or flight phase, but should be included in a composite rating or a flight 
phase rating to convey the realistic situation. It is preferable to evaluate 
separately such low probability occurrences as severe turbulence so that the 
pilot is not asked to weigh probability of occurrence and generalize a compos- 
ite rating to the point that it loses value. For example, a pilot may prefer 
to discuss a short duration occurrence of degraded handling qualities in the 
pilot comment data rather than to make a radical change in the task rating or 
flight phase rating. 

Such questions of probability of occurrence and levels of disturbances 
must be resolved as part of the mission description in the design of the exper- 
iment, with special attention often being required with respect to pilot 
orientation and the reporting of results. 


What the Rating Applies to 

Task and flight phase terminology .- For a pilot to evaluate the handling 
qualities of an aircraft, he must maneuver it and otherwise use it for its 
intended purpose. This purpose is given in a general sense by the role but 
more specifically by the designation of the required operations or mission. 
Next, it must be decided whether the handling qualities are being evaluated 
for a complete flight, a single flight phase, a subphase, or a specifically 
defined task. In any event, a task or series of tasks must be specified that 
will provide a suitable basis for pilot evaluation. The extent to which the 
evaluation task selected represents all aspects of the aircraft mission will 
depend upon several factors, the more important of which are program objective, 
simulation capability, and certain pilot considerations. From the definitions 
provided, it will be possible to establish conventional procedures for refer- 
ring to task ratings in contrast to flight, flight phase, or subphase ratings. 

General considerations.- To provide an overall pilot rating for a 
complete flight or aircraft role would likely involve so many situations that 
little of the information would relate to specific stability and control char- 
acteristics. For such a use, an overall rating would be of little value 
unless reference were made to the detailed comments provided by the pilot for 
each subphase. A flight subphase, on the other hand, is devoted to a single 
objective and has a single configuration or change in configuration. There- 
fore evaluation of a flight subphase would enable more direct correlation to 
be made between pilot rating and specific stability and control 
characteristics . 

Should significant changes then occur in either the aircraft state, the 
cockpit interface or the aircraft environment during a given flight phase, it 
might be desirable to confine the pilot rating to a flight subphase defined 
by a single aircraft state even though the task occurs in other subphases with 
different aircraft states. 

If an evaluation is to apply to a flight phase, the rating will tend to 
be weighted by the more adverse subphase characteristics. On the other hand, 
a longer task representative of one or more flight phases will provide a 
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better opportunity for the pilot to assess the workload involved, the 
adequacy of the information displayed, and the effect of unanticipated 
environmental disturbances. 

Short term maneuvers, particularly if determined to be the critical part 
of the task, will be most useful when the effect of specific stability and 
control parameters are assessed, but their use to the exclusion of a long-term 
mission-oriented task can reduce both confidence in and the fidelity of the 
handling qualities evaluation. The inclusion of auxiliary tasks in an evalua- 
tion task will depend upon the program objective, but may not be required for 
handling qualities studies in which the control task is of primary interest. 
Additional workload imposed by the auxiliary tasks can actually interfere with 
the pilot’s evaluation of the characteristics of greatest interest but, of 
course, must ultimately be taken into account. 

No hard and fast rule can be given for defining the evaluation task 
because it is obvious that the program objectives will play a significant 
part, but the nearer the selected task represents all the demands of the real 
mission during the flight phase or subphase being evaluated, the less will be 
the extrapolation required of the pilot. 

For pilot ratings that apply to a complete flight, flight phase, or 
subphase, the rating must consider all disturbances specified for the mission 
in the sense of being required operations. Thus, more extrapolation is 
expected for these ratings, placing greater reliance on the pilot's judgement 
than for either task or composite ratings. On the other hand, task and com- 
posite ratings must be reviewed with caution, as they may not include the most 
critical conditions and, therefore, could be misleading. 

The selection of a complete flight, flight phase, or subphase as the unit 
to be rated will therefore depend upon the program objective, the character- 
istics of primary interest, and the degree to which it is desired to draw upon 
the pilot’s training, experience, and knowledge to assess and extrapolate 
beyond the specific task provided. Several aspects of task selection involved 
in actual practice are discussed further in the following paragraphs. 

Task vs. flight phase or composite ratings .- To provide a rating based on 
the simulated task is, of course, the most direct approach and reduces the 
uncertainties of asking the pilot for extrapolation. It is weak, however, in 
that the task selected may not represent critical flight conditions, thereby 
allowing a nonconservative leniency in the interpretation of the pilot rating. 
Obviously, the pilot will not be as influenced by mission objectives if he is 
told to confine his rating to the evaluation task, although it may be diffi- 
cult for him to exclude considerations that he knows he would consider under 
actual operating conditions. If, however, all the separate tasks or subphases 
needed to define a flight phase can be provided, separate ratings may be 
obtained for each task or subphase, and a composite rating then given for the 
multiple tasks or subphases required to represent a flight phase. Providing 
all the separate tasks required to define a flight phase, however, is seldom 


20 



possible even by simulation. Nor is it always possible in actual flight, 
although its importance may dictate flying to remote areas to find 
environmental conditions for the actual tests. 

If it is then not possible to provide all the separate tasks required to 
define a given flight phase, it may be desirable to base a flight phase rating 
largely on the pilot's ability to draw on his knowledge, experience, and use 
of pilot-induced disturbances or self-induced tasks. Such a procedure 
utilizes the maximum capability of the pilot but also introduces extrapolation, 
with the inherent risk of his occasionally overlooking a potential problem or 
of being unduly critical of certain characteristics. The extent to which the 
pilot is expected to compensate for limitations in the simulated situation 
must be clearly understood at the start of a program. 

If the time factor and probability of occurrence are the same for each 
task, the composite rating will likely reflect the most critical task. If the 
probability of occurrence is low for one of the tasks (as might be attribut- 
able to severe turbulence), it is still possible for the probability of 
encounter to influence a composite rating in an undefined manner unless such 
a task is excluded from the composite rating. The experimenter must decide 
whether composite ratings, which include "probability of failure" or probabil- 
ity of encountering "unusual" environmental conditions, are to be assessed by 
the pilot or someone else. There must be no doubt as to whose responsibility 
this is, however. 

Transient disturbances.- When a steady-state failure is designated, the 
pilot is exempt from considering the probability of a failure. However, some- 
one must consider the probability of encountering transient disturbances from 
engine or other failures, wind shear or other short-term environmental condi- 
tions. In these cases, the pilot might have to consider the duration of the 
disturbance as well as its level. If specified in the mission definition, the 
pilot must consider disturbances whether they are provided in the task or not. 

Admittedly, a rating that reflects extrapolation for the effect of an 
undefined level and extent of a disturbance represents an approximation, but 
this may often be preferable to ignoring the disturbance. 

The effect of certain disturbances, such as those induced by the 
environment and the dynamics of failure, on pilot rating will depend heavily 
upon the aircraft state. As a rule, transient disturbances from whatever 
source will affect the pilot rating less if the handling qualities are good 
rather than poor. For this reason, it becomes increasingly important to pro- 
vide more accurate dynamic representation of transient disturbances as 
handling qualities deteriorate. 

Single axis or single parameter evaluations .- Confusion can occur when 
ratings are requested for specific parameters or for a single axis rather than 
for the complete pi lot- vehicle combination in performance of its mission. It 
should be handling qualities that are rated, with the influence of specific 
parameters identified through changes, if any, in the pilot ratings and from 
the pilot comment data. 
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Single-axis simulations can be of considerable value in quick look 
evaluations but should be clearly identified, so that the limitations of the 
data can be recognized. The task that, for example, includes freedom for 
maneuvering in pitch only cannot provide information on either total pilot 
workload or control coupling. When a single axis or single parameter is of 
primary interest, it is preferable to incorporate the single axis characteris- 
tics into a full multiple-axis (six degrees of freedom) task so that the 
evaluation can be made in the full context of handling qualities. 

Deficiencies evaluated in the longitudinal flight mode cannot be simply 
added to deficiencies evaluated separately in the lateral-directional mode. 

As a consequence, the most critical single axis task cannot necessarily be 
assumed to be the most critical overall pilot-vehicle task. 

A parallel situation arises in the general application of quantitative 
criteria for handling qualities. As a rule, such criteria are developed for 
specific parameters with all other parameters at a satisfactory level. Such 
is not necessarily the case in real life, so that one cannot expect to find 
single parameter criteria that apply accurately to all aircraft with their 
various roles, configurations, and operating environments. 

Preoccupation with variations in a single parameter should not be allowed 
to distract the pilot from the fact that his ratings apply to handling quali- 
ties as a whole, for the task (or flight phase) in context of the aircraft 
role. Assessment of single parameters or single axis effects can be very 
helpful to a designer, but special care must be taken in the development and 

presentation of such data. Programs of such limited scope are likely to 

introduce considerable pilot extrapolation with a resulting low level of con- 
fidence so that in some cases it may be more desirable to obtain only pilot 

comment data or comparative results not associated with a handling qualities 

rating. 

An evaluation may often involve comparisons or relative assessment of 
specific characteristics, but the evaluation pilot should not become so 
engrossed in comparisons that he loses sight of the absolute aspects of per- 
formance and workload in the context of the role and mission requirements. 

The use of an entirely separate rating scale, specifically for relative assess- 
ment of individual stability and control parameters, has been considered but 
abandoned because of possible confusion introduced by a second scale. 


Pilot Assessment Considerations 

Pilot rating .- As a shorthand representation of the handling qualities of 
an aircraft in the performance of a defined mission and task, a pilot rating 
will be meaningful only in proportion to the care taken in developing the pro- 
gram (defining objectives, the role and mission, the evaluation task, what the 
rating applies to, the simulation situation and extent of pilot extrapolation 
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involved). Unless a common basis is established and any criteria used are 
clearly indicated, one cannot expect to achieve reliable data and comparable 
ratings among pilots. Large disagreement between pilot ratings is usually 
traced to incomplete program development. 

There tends to be some disagreement among pilots as to how they actually 
arrive at a specific numerical rating. Some pilots lean heavily on the speci- 
fic rating description and look for the description that best fits their over- 
all assessment. Other pilots prefer to make the dichotomous decisions 
sequentially, thereby arriving at a choice between two or three ratings. The 
decision among the two or three ratings is then based upon the adjective 
description. In concept, the latter technique is preferable since it empha- 
sizes the relationship of all quality decisions to the aircraft role and 
mission requirements. With the final version of the revised scale, the pilot 
decisions shown must be considered in order to define the category boundaries. 

The actual technique used is probably somewhere between the two tech- 
niques discussed above and is not so different among pilots. In the past, the 
pilot 1 s choice has probably been strongly influenced by the relative useful- 
ness of the descriptions provided for the categories on one hand, and the 
numerical ratings on the other. The evaluation pilot is more or less continu- 
ously considering the rating decision process during his evaluation. He pro- 
ceeds through the dichotomous decisions to the adjective descriptors enough 
times that his final decision is a blend of both techniques. 

Half ratings (e.g., rating 4-1/2) generally indicate reluctance of the 
evaluation pilot to assign either of the adjacent ratings to describe the con- 
figuration. Any finer breakdown than half ratings is hardly ever justified 
since any number greater than or less than the half rating implies that it 
belongs in the adjacent group. Any distinction between configurations 
assigned the same rating must be made in the pilot comments. Use of the 
3-1/2, 6-1/2, and 9-1/2 ratings is discouraged because they represent impor- 
tant "boundary" conditions and the decision as to which category is selected 
may be significant even though hard to make. 

As a general rule, pilot ratings and comments are preferably given on the 
spot when the characteristics, performance, and workload are fresh in mind. 

If the pilot should later want to change his rating, the reasons for the 
change may be of interest. In some cases, an attempt should be made to repeat 
the configuration later in the evaluation program. 

Pilots' comments.- The use of a rating scale considered for universal 
handling qualities application leads to the assumption that the numerical 
pilot rating can represent the entire qualitative assessment. Extreme care 
must be taken against this oversimplification because the numerical ratings do 
not constitute the complete results of the data gathering process. 

As one might expect, the evaluation data most often neglected are the 
pilots' comments, either because they are not recorded or because they are 
often difficult to deal with because of their qualitative form, and perhaps 
their bulk. Often the nature of the experiment allows the pilot neither the 
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time nor the opportunity to identify the cause of his objections. Ratings, 
without the pilot's objections, are only part of the story. Only if the defi- 
cient areas can be identified, can one expect to devise improvements to elimi- 
nate or attenuate the shortcomings. The pilot comments are the means by which 
the identification can be made. 

The pilots' objections to the handling qualities are important therefore 
to the airplane designer who is responsible for improving the handling qual- 
ities and to the engineer who is attempting to understand and use the pilot 
rating data. If ratings are the only output, the engineer has no real way of 
assessing whether the objectives of the experiment were actually realized. A 
pilot's comments supply a means of assessing whether his objections (which 
lead to his summary rating) were related to the mission, to some extraneous 
factor in the execution of the experiment, or to his inaccurate interpretation 
of various aspects of the mission. For pilots' comments to be most useful, 
several details must be kept in mind. 

Generally, pilots should comment in the simplest possible language. 
Attempts to translate the observed characteristics or responses into engineer- 
ing terms should be avoided, unless such terms accurately supplement the 
pilot's observations given in descriptive terms. The pilot should report what 
he sees and feels, and describe his difficulties in carrying out whatever he 
is attempting. It is then important for the pilot to relate his difficulties 
in executing specific tasks to their effect on the accomplishment of the 
required operations. 

The pilot should be encouraged to make specific comments when evaluating 
each configuration. These comments generally are in response to questions 
developed during discussions of the mission and simulation situation. The 
pilot must also be encouraged to comment regarding his difficulties over and 
above the answers to the specific questions asked of him. In this regard, the 
test pilot should strive for a balance between a continuous running commentary 
and only occasional comment in the form of an explicit adjective. The former 
often requires so much editing to find the substance that it is often ignored, 
while the latter may add nothing to the numerical rating itself. 

The pilot’s comments must be collected during or immediately after each 
evaluation. If the comments are left until the conclusion of the evaluation, 
they are often forgotten. For both in-flight and ground simulator evaluations, 
this means that provision should be made for wire or tape recording. The 
immediate recording of key words or phrases on a kneepad can often be easily 
expanded to the full content after the task or evaluation is completed. It 
has been the experience of the authors that the best voluntary comments are 
often given during the evaluation. A useful procedure is to encourage volun- 
tary comment and note-taking during the evaluation and to require answers to 
specific questions in the summary comments at the end of the evaluation. 

Questionnaires or explicit check lists ensure that: (a) all important or 

suspected aspects are considered and not overlooked, (b) the reason for a 
given rating is specified, (c) an understanding is provided of the tradeoffs 
with which pilots must continually contend, and (d) supplementary comment that 
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Description Numeral 

No tendency for pilot to induce undesirable motions. I 

undesirable motions tend to occur wnen pilot inmates abrupt maneuvers 

or attempts fight control. These motions can be prevented or eliminated 2 

by puot techniques. 

undesirable motions ea$ ! iy induced when pilot initiates abrupt maneuvers 

or attempts tight control. These motions can be prevented or eliminated 3 

but only at sacrifice to 'as* performance or through considerable pilot 
attention and effort. 

Oscillations tend to develop when pilot initiates abrupt maneuver or 
attempts tight control. Pilot must 'educe gam or abandon task to recover. 4 

Divergent oscuations tend to develop when pilot initiates abrupt maneuvers 
or attempts tight control. Pilot must open loop by releasing or 5 

freezmq the stick. 

Disturbance or normal pilot control may cause divergent oscillation. Pilot 
must open control loop by releasing or freezing the stick. 

• These numerals are not related to the p^of rating scale. 


Figure 8.* Classification of PIO tendency. 


might not be offered otherwise is 
stimulated. It is recommended that 
the pilots participate in the prepar- 
ation of the questionnaires. The 
questionnaires should be modified if 
necessary as a result of the pilots 1 
initial evaluations. On occasion, it 
may be desirable to classify pilots* 
comments by having the pilots select 
one of several ranked comments about 
a specific characteristic. An exam- 
ple of such a classification of spe- 
cific pilots 1 comments is shown in 
figure 8 for PIO tendency. Classifi- 
cation as shown in this example is 
for easy identification only, and is 
not to be confused with the designa- 
tion for pilot rating. 


Simulation Situation 


A pilot evaluation is seldom conducted under the full circumstances of an 
actual flight. It almost always involves simulation to some degree because of 
the absence of the real situation. As an example, the evaluation of a day 
fighter is seldom carried out under the circumstances of a combat mission 
where the pilot is not only shooting at real targets but is being shot at with 
real bullets. Therefore, after the program has been defined and the decision 
made as to what is being rated, the relationship of the simulation to the real 
situation must be explicitly stated for both the engineer and the evaluation 
pilot so that each may clearly understand the limitations with respect to what 
is provided in the task, the environmental disturbances, the cockpit interface, 
and the completeness of the simulation of the stability and control 
characteristics . 

The pilot and engineer must know not only what is left out of an 
evaluation program, but also what is in that should not be in. The fact that 
the anxiety and tenseness of the real situation are missing, and that the air- 
plane is flying in the clear blue of calm daylight air instead of in the icy, 
cloudy, turbulent, darkness of the real mission could affect the results and 
should be considered during the program development and the interpretation of 
the results . 


Cockpit interface . - The cockpit interface must be specified, at least to 
the extent necessary to identify any factors that influence handling quali- 
ties. As a rule, once this is done for a given program it remains fixed and 
is merely reference information. When stability and control characteristics, 
for example, are the primary interest, features pertaining to the flow of 
information to and from the cockpit probably would not be considered as vari- 
ables in the study, but they must be noted with sufficient detail to establish 
at least any possible qualitative influence on the results. 
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The individual cockpit features, such as control system characteristics 
and instrument display, may, however, often include the specific variables to 
be studied. In some cases, these variables will be made known to the evalua- 
tion pilot, while in others, it will be more important that he assess only the 
overall handling qualities with respect to task accomplishment. 

Unless control system characteristics are a variable to be studied, it is 
desirable to provide an accurate representation of the anticipated vehicle 
control system. Such factors as control system break-out force, friction, 
inertial characteristics, or system gearing and sensitivity may significantly 
affect pilot rating. If it is not practical to represent the vehicle control 
system characteristics accurately then system characteristics should be pro- 
vided that are at least satisfactory to the evaluation pilots and do not 
introduce unwanted problems. It is often desirable to allow evaluation pilots 
to establish comfortable levels of control sensitivity, for example, before 
beginning an evaluation program. This procedure has distinct advantages in 
reducing the number of uncontrolled variables that may enter into an evalua- 
tion program. Care must be taken, however, that this procedure is not used 
when the control sensitivity is a variable. 

Repositioning any of the selectors available to the flight crew to change 
aircraft configuration will depend on the flight phase or task being evaluated. 
Visual information available to the pilot is defined by such features as the 
cockpit cut-off angle, the simulator system for providing external information, 
and the types and locations of the cockpit instruments. 

Aircraft environment.- The extent to which aircraft environmental factors 
are included in the simulation must be described. It is important that the 
level of such disturbances be related to the length of time they are to be 
coped with. This is usually for the duration of a flight phase, or specified 
task, but if it is to be considered as a transient condition, this should be 
noted . 

Should the evaluation be conducted under actual flight conditions, it may 
be desirable to secure a turbulence classification based on the standardised 
scale from the FAA Airman’s Information Manual (fig. 9), or to relate to other 
documentation as may be available. 

The probability of occurrence of various levels of turbulence is an 
important consideration, but is much too complex for the pilot to consider 
during an evaluation. 

The influence of environmental disturbances varies inversely with the 
level of handling qualities; that is, in general, the poorer the handling 
qualities, the more they are influenced by adverse disturbances. There is no 
simple rule, however, for translating ratings obtained in smooth air to 
ratings applicable to rough air. 

Environmental disturbances with some undefined probability of occurrence 
can strongly influence the aircraft design and its handling qualities. For 
this reason, their inclusion can be essential to the conduct of handling qual- 
ities experiments. When no provision is made for actual or simulated 
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^Intensity 

— 

Aircraft reaction 

Reaction inside aircraft 

Reporting term 

Definition ^ 


Turbulence that momentarily causes slight changes in aircraft 
attitude, altitude, or heading. Report as Light Turbulence: 

OR 

Occupants may feel a slight strain against 
seot belts or shoulder straps. 

Unsecured objects remain at rest, food 

Occasional 

Less than '/^ 
of the time 

LIGHT 

service may be conducted, and t-ttle or 

Intermittent 

y 3 to % 

Turbulence (light bumpiness! that causes slight aircraft 
fluctuations at rapid intervals without appreciable change m 
altitude, rol i, or yaw. Report as Light Chop. 

no difficulty is encountered in walking. 

Continuous 

More than 2 /3 


Turbulence that is similar to Light Turbulence but of greater 

Occupants feel definite strains against 

NOTE 


intensity. Changes in aircraft attitude, altitude, or heading 

seat belts or shoulder straps. 

!. Pilots should report locaton(s). 


occur, but the aircraft remains m positive control at all times. 

Unsecured objects are dislodged. 

time, ntensity, aUtude, type of 

MODERATE 

Report as Moderate Turbulence. 

OR 

Turbulence that is similar to Light Chop but of greater 
mtensity. It causes rapid bumps or jolts without appreciable j 
change in aircraft altitude, roll, or yaw. 1 

Report as Moderate Chop. 

Rood service ond walking are difficult. 

aircraft and, when applicable, 
duration of turbulence. 

2. Duration may be based on time 
between two locahons or over a 
single location. Ail locations should 
be readily identifiable. 

EXAMPLES 


Turbulence that causes large changes >n aircraft altitude, 

Occupants ore forced violently and 

a. Over Omaha, I232Z, moderate 


attitude, or heading. It may cause large variations in indicated 

repeatedly against seat belts or shoulder 

turbulence, Fligh 

Level 300, 

SEVERE 

airspeed. Aircraft may be momentarily out of control 

straps. 

B707 



Report os Severe Turbulence. 

Unsecured objects are tossed about, 
Food service and walking are impossible. 

b. From 50 miles south of 

Albuquerque to 30 miles north of 
Phoenix, I235Z to I250Z, 
occasional Moderate Chop, Flight 
Level 320, DC -8 

J 

EXTREME 

V 

Turbulence m which the aircraft is violently tossed about and 
is practically impossible to control. It may cause structural 
damaae. Report as Extreme Turbulence. 



Figure 9.- Turbulence criteria. 

environmental disturbances, it may be necessary for the pilot to learn their 
effect on handling qualities by employing self-imposed disturbances. Such 
procedures have limited application in a sophisticated simulation program, but 
do have considerable value in "short look" programs. 

Pilot stress.- Surprise and stress can interfere with the pilot's 
performance by distracting him from the primary task. Considering these fac- 
tors in depth is usually left to the more sophisticated programs, but those 
elements that contribute to greater fidelity and realism in a simulation pro- 
gram will be found desirable. The sharp jolt of the cockpit when a bad land- 
ing occurs contrasts vividly with the smooth conditions of a good landing. 

An element of surprise and stress should be introduced when the dynamic 
effects of control system or SAS failures are being evaluated. To be truly 
objective would require data from many occurrences introduced in a random man- 
ner during the time the pilot is engrossed in the performance of a task. 

If surprise and stress are not included in the task, the evaluation pilot 
must simulate surprise, distraction, or unattended operation by using a 
delayed reaction time. The stress or rapid increase in pilot workload when 
two or more adverse conditions or situations occur together can be studied in 
more sophisticated simulation programs. These potential "operating problems" 
consisting of combinations of failures or weather conditions are apt to raise 
the question of probability of occurrence. Again, this question should be 
separated from the pilot assessment whenever possible. The pilot rating 
should reflect the difficulty and effect on performance of the occurrence 
relative to successful completion of the task or flight phase under evaluation. 
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Therefore, such occurrences must originate with the program objective and be 
documented in the test plan as to how they will be executed; that is, whether 
they will be included in a task or left to the pilot’s individual assessment. 

Pilot extrapolation and confidence .- The question of simulation enters 
into nearly every evaluation program to some degree. Previous studies 
(ref. 9) have shown that sophistication is not necessarily the key to simula- 
tor usefulness although it can extend the range of application. Deciding 
M what a pilot rating applies to" (specific task or flight phase), and the 
completeness of the simulation will determine the degree to which pilot 
extrapolation is to be relied on. Neither the pilot nor the engineer retains 
confidence in the results if the need for extrapolation of observed results 
becomes too great. Guidelines have been drawn in this report to guard against 
this problem while at the same time encouraging full utilization of the 
trained and experienced test pilot. 

It is felt that careful planning and agreement on program objectives, 
mission definition, what is being rated, and the execution of the experiment 
can limit the uncertainties of extrapolation. 

In order to provide a means for expression, on the part of pilots in 
particular, a confidence factor could be introduced into programs in which 
simulation is involved. A definition might be "the ratio of the information 
available (to the pilot) in the simulation situation to the information 
required to derive a realistic pilot rating.” While provision is made for the 
use of such a confidence factor, no recommendation as to its use is given in 
this report. If used, care must be taken to insure that the confidence factor 
does not inhibit or otherwise confuse the actual pilot ratings assigned. As 
reference information, a confidence factor could help a pilot focus his com- 
ments on the adequacy of the simulation, but questionnaires could serve the 
same purpose. 

• CLASS A 

A pilot may usually assign a rating with a relatively high degree 
of confidence, although be may have mild reservations because of 
incomplete or inadequate simulation of motion cues, disturbances, visual 
information, or other factors affecting pilot workload. 

Supplementary tasks, if needed, con be adequately provided by 
the pilot. 

• CLASS B 

A pilot can assign a rating with only a moderate level of 
confidence because of uncertainties introduced by a lack of representative 
environmental disturbances as well as incomplete or inadequate simulation 
of motion cues, disturbances, visuol information, or other foctors 
affecting pilot workload 

Supplementary tasks may be desired, but are not available. 

• CLASS C 

A pilot can assign a rating with only minimum confidence because 
considerable pilot extrapolation is required due to an incomplete task, 
thereby requiring considerable reliance on self-imposed tasks and 
maneuvers for assessment. 

This may also be aggravated by incomplete or very limited 
simulations of motion cues, disturbances, visual information, or 
other factors affecting pilot workload. 


For those who favor using a 
confidence factor, a simple classi- 
fication is proposed in figure 10, 
as A, B, or C. Studies involving 
"quick look" or rudimentary simula- 
tion may often call for pilot rat- 
ings in which the pilot is confident 
in only a relative rather than an 
absolute sense. For these, a "C" 
classification might be appropriate. 
An attempt should be made to place 
all other evaluations in Class A, 
thereby allowing Class B to be used 
only when a pilot has difficulty 
deciding between A and C. 


figure 10.- Classification of pilot confidence factor. 
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The Briefing Guide 


In order to summarize the content of this report in a form which may be 
applied directly to the execution of handling qualities experiments, a 
"Briefing Guide" has been assembled in appendix B. The purpose of this guide 
is to outline a format to insure that all pertinent documentation is covered 
for each evaluation program. Only in this way can the evaluation pilots be 
sure that everyone is talking about the same thing. Subsequent analysis of 
the data, reporting, and ultimate comparison with data from other sources is 
then materially aided. 

The briefing guide enables the pilot and engineer to know what is missing 
from an evaluation program, as well as what is provided. 

Once the information is tabulated, it should be apparent just what is 
being evaluated and rated, what the task is, the conditions under which the 
evaluation is performed, and the applicability of the data. Uith the impor- 
tant information provided in a Briefing Guide, it should be possible to 
evaluate handling qualities more effectively and to improve communication 
not only between pilots and engineers, but between various research and 
development groups as well. 


Ames Research Center 

National Aeronautics and Space Administration 

Moffett Field, Calif. 94035, Nov. 13, 1968 
125-19-06-06-00-21 
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APPENDIX A 


COMPENSATION - 


CONFIGURATION - 


COCKPIT INTERFACE - 


CONTROLS - 


Primary - 


Secondary - 


Selectors - 


DEFINITIONS 


The measure of additional pilot effort and attention 
required to maintain a given level of performance in the 
face of deficient vehicle characteristics. 

The aircraft geometry as established by the actual 
position of movable portions and surfaces controllable 
by the selectors, and the state of operability of 
on-board systems. 

Examples: Auto-throttle on, flaps at 15°, wing sweep 

at 45°, etc. 

The means provided for the flow of information to and 
from the pilot. These include the display of informa- 
tion available to the pilot as well as the type and 
characteristics of the cockpit controls. 

Examples: Description of cockpit instruments and layout. 

Description of control system (i.e., stick, 
wheel, force - deflection gradients, sensi- 
tivities, primary, secondary controls, and 
selectors provided) . 

A distinction is made between the types of controls in 
the cockpit according to their function. Principal con- 
trols are the primary and secondary controls . 

Those controls used by a pilot to continuously modify 
the movement of the aircraft. 

Examples: Pitch, roll, yaw controls, throttle, DLC . 

Those controls used by a pilot to make discrete changes 
in the movement or balance of the aircraft, thereby modi- 
fying the need for actuation of the primary controls . 

Examples: Pitch, roll, and yaw trimmers, aerodynamic 

braking devices. 

Those cockpit controls available to the crew for 
changing aircraft configuration. 

Examples: Flaps, slats, wing sweep, BLC . 
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FAILURE STATE 


FLIGHT or SORTIE - 


FLIGHT PHASE - 


FLIGHT SUBPHASE - 


HANDLING QUALITIES - 


A steady-state failure characterized by the various 
failed systems that affect the handling qualities. The 
dynamic effect of a failure is called a change of state 
and should be noted separately. 

Examples: Any failure resulting in loss of selected 

function. Engine failure, augmentation sys- 
tem, failure in stability, autothrottle, pri- 
mary flight control system (power boost, 
electric stick, servo control feel, etc.) or 
secondary flight control system (trim, aero- 
dynamic brake, etc.). 

A complete sequence of flight phases of an aircraft 
within one of its roles. Full or complete mission. 

Example: The composite of takeoff, climb, cruise, com- 

bat (or other special phase), descent, 
approach, landing. 

A designated portion or segment of a complete flight. A 
mission phase. A flight phase may be represented by one 
or more separate tasks. 

Examples: (a) Common phases - takeoff, climb, cruise, 

descent, approach, and landing. 

(b) Special phases required by role - 
formation, refueling, air-to-air or air-to- 
ground combat, weapon delivery, emergency 
conditions (i.e., 2- or 3-engine operation, 
emergency descent, etc.), VTOL transition, 
VTOL hover, STOL takeoff, and STOL approach. 

That part of a flight phase having a single objective, 
and a single configuration or change in a configuration. 

Examples: Air-to-air tracking, terminal area holding, 

glide slope capture, localizer capture, ILS 
tracking , wave -off . 

Those qualities or characteristics of an aircraft that 
govern the ease and precision with which a pilot is able 
to perform the tasks required in support of an aircraft 
role . 


31 



MANEUVER - 

MISSION - 
PERFORMANCE - 

ROLE - 

SPECIAL CONDITIONS - 

STATE - 
TASK - 

Control - 


A planned and regulated movement of an aircraft for the 
purpose of aiding the completion of a given control task 

Examples: Bank, turn, dive, pullup, turn reversal, roll 

reversal, rolling pullup, steady sideslip, 
return from sideslip, control steps and 
pulses, maintenance of a steady condition. 

The composite of pilot-vehicle functions that must be 
performed to fulfill operational requirements. May be 
specified for a role, complete flight, flight phase, or 
flight subphase. 

The precision of control with respect to aircraft move- 
ment that a pilot is able to achieve in performing a 
task. (Pilot-vehicle performance is a measure of 
handling performance. Pilot performance is a measure 
of the manner or efficiency with which a pilot moves 
the principal controls in performing a task.) 

The function or purpose that defines the primary use of 
an aircraft. 

The special circumstances pertinent to the evaluation 
(i.e., aircraft environment and pilot stress). 

Examples: Special conditions of weather and environment 

turbulence, wind shear, ceiling, visibility - 
night, etc. Pilot awareness, surprise, or 
distraction with respect to impending failure 
or disturbances. 

The mass distribution and failure situation that deter- 
mine completely the behavior characteristics of the 
aircraft. A state without a failure is a normal state. 

The actual work assigned a pilot to be performed in 
completion of or as representative of a designated 
flight segment. 

That part of a task which requires continuing actuation 
of the principal controls and use of the selectors (see 
"CONTROLS") as required. 

Examples: Movement between specified points, tracking 

part of weapon delivery, ILS or VOR tracking. 
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Auxiliary - 


WORKLOAD - 


Physical - 


Mental - 


That part of a task which involves the pilot in actions 
other than direct control of the aircraft. 

Examples: Navigation, communication monitoring, and 

selection of systems. 

The integrated physical and mental effort required to 
perform a specified piloting task. 

The effort expended by the pilot in moving or imposing 
forces on the controls during a specified piloting task. 

Mental workload is at present not amenable to quantita- 
tive analysis by other than pilot evaluation, or indi- 
rect methods using physical workload (input) and the 
task performance measurements. An example would be the 
improvement associated with flight-director type dis- 
plays which reduce the mental compensation normally 
required of the pilot. 
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APPENDIX B 


BRIEFING GUIDE AND RATING INFORMATION FOR HANDLING QUALITIES EXPERIMENTS 

PROGRAM DEFINITION 

Objectives 

General : 

Research Q Development f~| Acceptance Q Other f~[ 

Aircraft role 

Flight segment of interest 

Parameter or variable of primary interest 

Mission description: 

General statement of the required operations for the flight segment 
of interest in context of aircraft role. 

1. What is pi lot- vehicle combination required to accomplish? 


2. What are the conditions under which these required operations are 
to be carried out (i.e., aircraft state, environment, and cockpit 
interface) ? 
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Scope 

Short look (guide to further tests) I I 
Long look (definitive as possible) | | 

Task related only (minimize pilot extrapolation) [~| 

Mission or flight segment related (pilot extrapolation encouraged) [_] 
Nature of task or tasks provided: □ 

Short term Q Long term I I 

Critical task(s) included? O 0r left for pilot extrapolation? □ 
Identify : 


Are simulated disturbances provided or to be supplied by pilot? 


Use of pilot-initiated or self-imposed tasks, disturbances, or short- 
term maneuvers . 


Provision for familiarization: 


Measured performance: 

Is performance to be measured? 
Pilot-vehicle (output) [~] 

Pilot workload (input) | | 
Other measurements? □ 
Additional provisions: 
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Are additional runs allowed for evaluation? . Explain: 


RATING INFORMATION 


What is to be rated? 

[^Handling qualities | | Other 

I I Task 

I I Flight segment (flight phase or subphase) 

□ Composite of tasks (specify tasks included in composite rating) 

I I Composite of subphases (specify subphases) 

What is provided? 

Test vehicle: (aircraft, spacecraft, simulator, etc.) 


Aircraft state: 

Normal Q Emergency Q Identify failure: 


Configuration : 


Gross weight: 

Mass distribution : 
Changes in aircraft state: 
Configuration changes 


(during task or designated phase) 
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Transient failures (unanticipated failures to be introduced into task 
for pilot reaction to and correction of resulting disturbance) 


Cockpit interface : 

(Brief description of important items noting unusual or detracting or 
limiting characteristics in particular) 


Principal controls : 
Primary controls : 


Secondary controls : 


Selectors : 

Cues and disturbances provided: 


Motion : 


Visual : 

Instruments: Conventional Q Novel I I 

Full panel O Part panel | 1 

(Brief statement of instruments or characteristics which may 
affect the evaluation) 
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External vision: 


External visual display VFR provided □ not provided □ 

IFR only □ 

(Brief description of information provided) 


Task (s) : 

Control task description: 


Auxiliary tasks? Yes Q No I I 

Aircraft environment : 

Day □ Night Q 

Visibility conditions : 


Turbulence Q] 

Wind shear 

Crosswind O 

Other weather conditions: 
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PILOT EVALUATION DATA 


Form of results 

Pilot rating Q Comments | | Questionnaire Q Oral debriefing | | 
Other | | 


Pilot rating scale: 

Principal items for comment: 


Program assessment: 

(Comments and recommendations on validity, effectiveness, etc. 
Confidence factor - if used) 
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APPENDIX B - Example 1 


BRIEFING GUIDE AND RATING INFORMATION FOR HANDLING QUALITIES EXPERIMENTS 


PROGRAM DEFINITION 


Objectives 

General : 

Research □ Development S Acceptance □ Other □ 

Aircraft role Commercial transport 

Flight segment of interest Landing approach. 

Parameter or variable of primary interest Longitudinal stability^ 
Mission description: 

General statement of the required operations for the flight segment 
of interest in context of aircraft role. 

1. What is pilot -vehicle combination required to accomplish? 

Con duct VFR and IFR approaches and landings, either Day or Night , 

with an accuracy ±50 ft from. d&sired^ pafrfo crwti ±5 k airspeed^ Ninety^ 
percent consistency is allowable if pitot can execute wave off sajeiy 
or ma ke an acceptable landing^ ( 5 ft/sec at touchdown within SO ft 
of centerline , with side velocity less, than 5 ft/sec ) 


2. What are the conditions under which these required operations are 
to be carried out (i.e., aircraft state, environment, and cockpit 
interface) ? 

Both normal and eme rgency stade^ with pitch SAS failed^ Approach 

config uration 3 Light-M 'oderate turbulence t Night or Day , 0-20. k 

crosswi nd , 10 k per 100 ft winds hear. Cockpit interface 

ite ms mu st not be considered limktt of to introduce. signifi cantly 
objectionable characteri sties to tlw evaluation pilots^ 
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Scope 


Short look (guide to further tests) 0 
Long look (definitive as possible) S 
Task related only (minimize pilot extrapolation) CD 

Mission or flight segment related (pilot extrapolation encouraged) (X) 
Nature of task or tasks provided: 

Short term CD Long term IXl 

Critical task(s) included? CD Or left for pilot extrapolation? iXl 
Identify: Consider effect of higher level of turbulence on instrument 
approach « 


Are simulated disturbances provided or to be supplied by pilot? 

Light turbulence and wind shear provided. 

Use of pilot-initiated or self-imposed tasks, disturbances, or short- 
term maneuvers . 

During familiarization runs only » 


Provision for familiarization: Five runs at beginning of each 

session , 

Measured performance: 

Is performance to be measured? Yes 

Pi lot- vehicle (output) [Xj ILS tracking error 

Control column and. throttle activity 
Pilot workload (input) [X| (reversals and work) 

Other measurements? (X) Altitude and airspeed error 

Additional provisions: All but familiarization runs used only to 
verify learning curve , 
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Are additional runs allowed for evaluation? Yes . Explain: 

Pi tot may make Taonperfo^^ evaluation runs as tijne allows^ after 
oom^letion of alt data vuns_. 


RATING INFORMATION 


What is to be rated? 

® Handling qualities 
□ Task 


I I Other 


Flight segment (flight phase or sub phase) Landing approach 
I I Composite of tasks (specify tasks included in composite rating) 


Composite of subphases (specify subphases) 

What is provided? 

Test vehicle: (aircraft, spacecraft, simulator, etc,) S-16 Moving 

B ase Simulator 

Aircraft state: 

Normal D Emergency [X) Identify failure: Pitch SAS failed . 

Configuration: Approach configuration 3 gear I ,, flap I 40° leading 

edge extended , wing sweep 20° 3 etc . 

Gross weight: Maximum landing weight 

Mass distribution: C.G. at aft limit 

Changes in aircraft state: 

Configuration changes (during task or designated phase) 

Flap deflection increased from maneuver to approach upon 

intercepting glide slope . 
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Transient failures (unanticipated failures to be introduced into task 
for pilot reaction to and correction of resulting disturbance) 

Pitch SAS failure to he introduced unexpectedly on two SAS on runs . 

Cockpit interface : 

(Brief description of important items noting unusual or detracting or 
limiting characteristics in particular) 


Principal controls : 

Primary controls: Conventional wheel and yoke with 707 feel 
characteristics ; single-throttle control for all engines . 


Secondary controls: Electric trim switch on wheel for pitch and 
roll 3 yaw trim - not provided; spoiler actuation not provided in 
cockpit j but available on voice command. 

Selectors : 

Cues and disturbances provided: Pitch and roll cues with turbulence 

in vertical motion. 

Motion: 


Visual : 

Instruments: Conventional (X) Novel 0 

Full panel 0 Part panel SO 

(Brief statement of instruments or characteristics which may 
affect the evaluation) 
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External vision: 


External visual display VFR provided 13 not provided □ 

IFR only □ 

(Brief description of information provided) 

Closed circuit color T.V. used to provide realistic visual cues 
1500 ft to ground accurate 14° vertical (downward) cutoff angle 
provided y but lateral view restricted to 40° angle . Prominent 
horizon provided . Fairly good picture resolution enables 
Z ft/sec touchdown to be made consistently y but not good enough 
for much less . 

Task (s) : 

Control task description: a. Visual approach from crosswind leg y 
7000-ft altitude at recommended approach speed to land at 5000-ft 
elevation . b. IFR-ILS approach 3 capture and track 3° ILS from 
OM . Flo offset crosswind or turbulence 2000 ft RVR . <?. Same as 

b. y except with 20-k crosswind , light turbulence y moderate 
windshear at 100 ft above ground . 2000 ft RVR 

Auxiliary tasks? Yes □ No (3 
Aircraft environment : 

Day IS Night S 

Visibility conditions : Test variable 


Turbulence S Light turbulence 
Wind shear 


10 k per 100 ft (ZOO ft - 50 ft altitude) 


Crosswind IX] 20 k 


Other weather conditions: None 
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PILOT EVALUATION DATA 


Form of results 

Pilot rating ® Comments [D Questionnaire [U Oral debriefing (X) 
Other [I] 


Pilot rating scale: Cooper-Harper 

Principal items for comment: Airspeed and flight-path control . Pitch 
response . Pitot workload . 

Program assessment: 

(Comments and recommendations on validity, effectiveness, etc. 
Confidence factor - if used) 

Validity of task , Adequacy of display . 
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APPENDIX B - Example 2 


BRIEFING GUIDE AND RATING INFORMATION FOR HANDLING QUALITIES EXPERIMENTS 

PROGRAM DEFINITION 

Objectives 

General : 

Research (X) Development CD Acceptance CD Other CD 

Aircraft role Long-range attack bomber 

Flight segment of interest In-flight refueling 

Parameter or variable of primary interest Lateral control 

Mission description: 

General statement of the required operations for the flight segment 
of interest in context of aircraft role. 

1. What is pilot-vehicle combination required to accomplish? 

By visual references > attain and maintain position relative to 

tanker within -presented bounds for period of 10 min . 


2. What are the conditions under which these required operations are 
to be carried out (i.e., aircraft state, environment, and cockpit 
interface) ? 

V - 290 KIAS 3 h = 30 3 000 ft 3 W - initial and refueled. Daytime 3 

smooth air and light turbulence . Straight flight and turn entries . 
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Scope 


Short look (guide to further tests) I I 
Long look (definitive as possible) [X] 

Task related only (minimize pilot extrapolation) ED 

Mission or flight segment related (pilot extrapolation encouraged) lX) 
Nature of task or tasks provided: 

Short term ED Long term (X) 

Critical task(s) included?® Or left for pilot extrapolation? ED 

Identify : Success or failure to establish and maintain position with 

respect to tanker with light turbulence and occassional disturbance 

of the tanker . 

Are simulated disturbances provided or to be supplied by pilot? 

Provided 

Use of pilot-initiated or self-imposed tasks, disturbances, or short- 
term maneuvers . 

Only during familiarization or subsequent to notification of test 

director . 

Provision for familiarization: As desired by pilot . 


Measured performance: 

Is performance to be measured? Yes 
Pilot-vehicle (output) lXl 


Pilot workload (input) ED 
Other measurements? ED 
Additional provisions: 
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Are additional runs allowed for evaluation? Yes . Explain: 


RATING INFORMATION 


What is to be rated? 

(X) Handling qualities Othe r of sirnul '.ation. 

□ Task 

1x1 Flight segment (flight phase or subphase) Special phas<3 - refueling . 
CD Composite of tasks (specify tasks included in composite rating) 

□ Composite of subphases (specify subphases) 

What is provided? 

Test vehicle: (aircraft, spacecraft, simulator, etc . ) 

Aircraft state: 

Normal (X) Emergency CD Identify failure: 

Configuration: Refueling 


Gross weight: 200^000 and 400 3 000 Zb 

Mass distribution: Appropriate 

Changes in aircraft state: 

Configuration changes (during task or designated phase) 
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Transient failures (unanticipated failures to be introduced into task 
for pilot reaction to and correction of resulting disturbance) 

Yam damper 

Cockpit interface: 

(Brief description of important items noting unusual or detracting or 
limiting characteristics in particular) 


Principal controls: 

Primary controls: Contr ol stick with light control forces^ good. 
center ing and low friction . Convent ional rudder. pedals. having. 
light forces but objecHonaiyly poor centering. Requires. pilot 

adaptation and special c onsuiervction. 

Secondary controls: Trim not provided^ 


Selectors : Not provided - core/iffuratfon unchanged. 

Cues and disturbances provided: 

Motion: Pitch, roll , and you) (±45°). Acce lerati ons attenuated. by 

l imited trcmslation (±5 ft vertically } ±10 ft laterally^ 

±10° fore and aft). 

Visual : 

Instruments: Conventional ® Novel CD 

Full panel CD Part panel CD 

(Brief statement of instruments or characteristics which may 
affect the evaluation) 

Cons i d ered primarily a visual task. Only primary flight 

ins truments - airspeed , altitude 3 and rate of climb provided. 
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External vision: 


External visual display VFR provided (X) not provided I 1 

IFR only I I 

(Brief description of information provided) 

Closed circuit T.V., enabling visual formation flying with 
stationary model of tanker aircraft . 


Task (s) : 

Control task description: (1) Close on tanker ^ establish contact y 

maintain hookup for duration of refueling operation without 

turbulence. (2) Same as (1), except for simulated light 

turbulence and occasional abrupt displacement of tanker to 

simulate disconnect. 


Auxiliary tasks? Yes Q No [XI 
Aircraft environment: 

Day IS Night CD 

Visibility conditions: Unobstructed - except for accurate 
windshield and visibility representation . 

Turbulence (XJ Light 
Wind shear I I 
Crosswind CD 


Other weather conditions: 



PILOT EVALUATION DATA 


Form of results 

Pilot rating [X] Comments Q Questionnaire [X) Oral debriefing D 
Other O Questionnaire will be made available after completion of test . 


Pilot rating s cale : Cooper-Harper 

Principal items for comment: Pilot workload. Influence of control 

system characteristics 3 if any . 

Program assessment: 

(Comments and recommendations on validity, effectiveness, etc. 
Confidence factor - if used) 

Realism of task . Adequacy of simulated motion . 
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